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A Synthesis of Evaluation Studies 

Compiled by the staff of the California Collaborative on District Reform 

This document synthesizes the findings from the evaluations of four statewide programs 
designed to provide support for underperforming schools and districts in California as part of 
the state accountability system: 

• Immediate Intervention/Underperforming Schools Program (II/USP) 

• High Priority Schools Grant (HPSG) Program 

• School Assistance and Intervention Teams (SAIT) 

• District Assistance and Invention Teams (DAIT) 

This overview outlines the cross-cutting findings and recommendations from the evaluation 
studies of these programs. In addition, this document includes two charts. The first provides 
programmatic details for each effort, including timeline, school or district selection criteria, state 
funding for implementation, and additional technical support provided. The second chart 
summarizes the evaluations of each program, including study design and findings related to 
student performance outcomes as well as implementation. 

Study Findings 

Mixed methods studies were conducted to evaluate the impact of II/USP, HPSG, SAIT, and 
DAIT on student outcomes. Without exception, the programs either had negligible or no overall 
program effects on student achievement. The majority of the findings from the evaluation 
studies focused on program implementation. School and district staff generally reported that 
their program participation had positive effects on school capacity, instruction, and data-based 
decision-making. This was true across all four programs. In two of the four studies, participants 
reported that the school or district relationship with their external provider helped improved 
principal capacity. In at least two of the studies, an increased focus on monitoring student 
achievement data was associated with improved student performance, and qualitative responses 
to surveys and interviews in all four studies mentioned that schools and districts had increased 
their use of data to inform instruction or policy. 

Study Recommendations 

Each of the evaluation studies presented a series of recommendations to improve the processes 
through which the state supports underperforming schools and districts. While many 
recommendations were program-specific, there were three general themes that emerged. In at 
least three of the four studies, the authors made the following recommendations for program 
improvement: 


1 




• Provide additional funding for sustainability; 

• Enhance the role of the district to support school improvement; and 

• Increase monitoring and accountability expectations. 

The need for additional financial resources was the most frequent recommendation, appearing 
all four evaluation studies. 



Overview of Improvement Programs 


Program 

Timeline 

School/District Selection 

Funds Provided 

Technical Assistance Provided 

Immediate 
Intervention/ 
Underperforming 
Schools Program 
(II/USP) 

1999-00 

through 

2004-05 

Cohort 1 (430 schools): Schools 
scoring in the bottom half of the 
state’s schools on the Stanford 
Achievement Test, 9th Edition 
(SAT-9) for two consecutive years 
(1998 and 1999) were invited to 
submit an application to 
participate in the program 

Cohorts 2 & 3 (860 schools, 430 
each in each cohort): Schools had 
Academic Performance Index 
(API) scores in the lower five 
deciles and had not met their API 
growth targets in the previous 
year 

$50,000 for the first year to 
develop an Action Plan for 
school improvement, then 
$200 per pupil per year to 
implement the Action Plan 

II/USP schools received funds to create and implement an Action Plan for 
school improvement with the assistance of a state-approved External 
Evaluator. 

The External Evaluator was responsible for assisting in the creation of 
the Action Plan, particularly in supporting the involvement a variety of 
stakeholders during plan development. 

The External Evaluator was also responsible for supporting and 
monitoring the implementation of the Action Plan. 

High Priority 

Schools Grant 
(HPSG) Program 

2002-03 

through 

2005-06 

A total of 658 schools statewide 
participated in the program. 

While schools in deciles 1-5 were 
eligible, priority for participation 
in the HPSGP was given to 
schools ranked in the lowest 
decile on the state API. 

$50,000 for the first year to 
develop the Action Plan for 
school improvement, then 
$400 per pupil per year for 
three years (and possible 
fourth year) to Implement 
Action Plan 

Schools were required to develop an Action Plan (or modify an existing 
plan) to serve as a blueprint for the school and community to focus on 
improving student achievement and meeting growth targets. 

Districts monitored the development of the Action Plan and reported on 
schools participating in the HPSGP by submitting annual reports 
accounting for school characteristics such as instructional materials 
used, courses offered, levels of parental involvement, teacher training, 
and principal experience. 

School Assistance 
and Intervention 
Teams (SAIT) 

2003-04 

through 

2009-10 

Schools were placed in state 
monitoring status when, after 
participation in either II/USP or 
HPSGP, they did not make 
significant yearly API growth. 

All state-monitored schools were 
required to obtain the services of 
an SAIT provider. 

Each cohort started with 430 
schools 

Districts were provided one¬ 
time funds for each state- 
monitored school: $75,000 for 
each elementary and/or middle 
school and $100,000 for each 
high school 

The SAIT provider was responsible for assessing the degree to which the 
school was implementing nine Essential Program Components: adoption 
of SBE-adopted or standards-aligned instructional materials; adequate 
instructional time; participation of in School Administrator Training 
Program; engagement of highly qualified teachers and universal 
participation of ELA and math teachers in state Professional 

Development Program; implementation of a student achievement 
monitoring system; ongoing instructional assistance and support for 
teachers; monthly teacher collaboration facilitated by the principal; use 
of lesson and course pacing schedule and master schedule flexibility; and 
district alignment of fiscal support. 

Working with the District/School Leadership Team (DSLT) and school 
staff, the SAIT provider completed an Academic Program Survey to 
determine the school’s existing level of implementation of the nine EPCs. 
Based on the information collected during this assessment, the SAIT 
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Program 

Timeline 

School/District Selection 

Funds Provided 

Technical Assistance Provided 





provider produced the Report of Findings and Corrective Actions (ROF) 
and worked collaboratively with district and school staff to establish 
benchmarks for each corrective action. The SAIT provider was required 
to complete the ROF within 30 days of being engaged. The local Board of 
Education was required to review and approve the ROF within 30 days. 
Once the local Board approved the ROF, it was to be submitted to the 

CDE to become the basis for revising and aligning the school plan to 
better implement all of the EPCs. 

District Assistance 
and Invention 

Team (DAIT) 

2008- 09 
through 

2009- 10 

LEAs identified as not making 
adequate yearly progress (AYP) in 
accordance with Title 1 legislation 
for three or more years fell/fall in 
to Program Improvement (PI) 
status and were targeted for DAIT 
support. 

LEAs in "severe" need of assistance 
were required to contract with a 
specific district assistance and 
intervention team (DAIT), 
selected for them by the state. 

LEAs in "moderate 11 need of 
assistance were required to select 
and work with a DAIT of their 
choosing. 

43 districts received the DAIT 
intervention 

California spent $44.25 
million on the 43 districts that 
received the DAIT 
intervention over the 2008-9 
and 2009-10 school years, or a 
little over $1 million per 
district that received the 
intervention. Elowever, 
amounts per district ranged 
from $200,000 to $4.8 million. 
California ranked and grouped 
the PI3, districts (intensive, 
moderate and light) and 
provided differentiated 
funding based on criteria 
adopted by the 

California State Board of 
Education. 

LEAs received support from a state-approved DAIT provider who 
assisted them in diagnosing district needs and summarizing their 
findings in a capacity study. DAIT providers also helped LEAs develop or 
rewrite an existing improvement plan to incorporate recommendations 
from the capacity study. 

The DAIT provider also supported LEA efforts to implement and monitor 
their progress on improvement plans. 
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Overview of Program Evaluation Studies 


Program & Study 

Design 

Findings - Student Outcomes 

Findings - Implementation 

Proaram: 

Immediate 
Intervention/ 
Underperforming 
Schools Program 
(II/USP) 

Study: Bitter, Perez, 
Parrish, Gonzalez, 
Socias, Salzfass, &r 
Esra, 2005. 

American Institutes 
for Research. 

Mixed-methods: 
impact analysis of 
statewide achievement 
trends using data from 
all II/USP schools and 
relevant comparison 
schools; interviews 
with district- and 
schoohlevel 
administrators and 
teachers in II/USP 
schools growth. 

The overall program effect was negligible. Any small 
advantage experienced by II/USP schools relative to 
comparison schools during program participation 
dissipated before or soon after program completion. 

There was evidence that II/USP participation contributed to growth 
in some schools. Even though there was no overall impact of II/USP on 
student achievement, some previously struggling schools were able to 
make substantial progress during the time of II/USP implementation, and 
some attributed this growth to participation in the program. 

• Facilitating factors: Several essential factors for growth were 
identified by schools that made consistent and/or high growth 
in student achievement during II/USP. These included school 
capacity, instructional coherence, and systematic assessment 
and data-based decision-making. 

Local districts were found to influence the achievement trends in 
low-performing schools. While district supports were reported as key 
to some schools’ improvement, these supports did not appear to be 
present in all schools. 

Differing state and federal accountability systems diffused the 
attention schools were paying to the Public Schools Accountability 
Act. 

Proaram: Hiah 
Priority Schools 

Grant (HPSG) 
Program 

Study: Harr, 

Parrish, Socias, & 
Gubbins, 2007. 
American Institutes 
for Research. 

Mixed-methods: 
impact analysis of 
extant student' and 
schoohlevel 
achievement data for 
HPSGP and non- 
HPSGP schools; 
Conducted case studies 
of 16 HPSGP schools; 
Administered and 
analyzed data from 
surveys in 106 HPSGP 
schools and 49 
districts. 

There was no overall program effect. While HP Only 
plus planning schools demonstrated academic progress 
during the period of program implementation, their gains 
were not statistically different from the gains of the 
comparison schools included in these analyses. 

Many survey respondents were not aware of program requirements. 
While nearly half of the school survey respondents described their 
external provider support as appropriate and effective, nearly 45 percent 
of school staff reported that the school did not use, or reported that they 
did not know if the school used, an external provider in the development 
of the Action Plan, 

Half of survey respondents were concerned about the short length of 
the program. Although the majority of HPSGP school respondents 
indicated an effective use of funds, half reported concern about the short 
length of the program, and nearly a third reported that the untimely 
arrival of funds did not permit appropriate planning and spending. 

Spending on personnel was reported as the most common and the 
most effective local use of HPSGP funds. 

A slight majority of respondents reported a lasting HPSGP impact in 
areas of school capacity. 

Survey respondents had mixed opinions on the sustainability of 
program impacts. While 60 percent of school respondents indicated 
confidence in sustaining the impact of HPSGP, only 40 percent said they 
had been able to find funding to continue these reforms. 
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Program & Study 

Design 

Findings - Student Outcomes 

Findings - Implementation 

Program: School 
Assistance and 
Intervention Teams 
(SAIT) 

Study: McCarthy, 

Li, Tabernik, & 
Casazza, 2008. 
Hatchuel Tabernik 
and Associates. 

Mixed methods: 
Longitudinal analysis of 
student achievement 
(API and STAR) data 
from 200 schools 
working with SAIT 
providers; qualitative 
reports on schools’ 
progress in 
implementing the 

EPCs; analysis of online 
surveys of SAIT 
providers, district 
personnel, principals, 
and teachers; and visits 
to SAIT schools. 

Close analysis of the API and STAR data reported in 
this evaluation do not sufficiently support the authors’ 
conclusion that participation in SAIT improved student 
learning outcomes (as measured by API and CST 
scores) at a faster rate than was the case for non-SAIT 
decile 1-5 schools. Both Cohort 1 and Cohort 2 SAIT 
schools performed at or near the average of for all API 
decile 1-5 schools the year prior to identification for SAIT; 

3 or 4 years later, they scored near or slightly below the 
non-SAIT average. The trend of declining achievement 
prior to selection into SAIT appears in some cases to be a 
one-year perturbation, making interpretation of 
subsequent increases difficult. 

Results for particular subgroups varied by subgroup, SAIT 
cohort, and subject area. Elowever, the African-American 
sub-group consistently did not achieve statistically 
significant increases in ELA and mathematics compared to 
their pre-SAIT performance. 

The majority of site visit respondents believed that SAIT had 
improved their school. 

Actions perceived by respondents as facilitating successful 
implementation included: 

• Hands-on leadership by the principal 

• Assessing students, identifying their needs, and placing them in 
appropriate classes early in the school year. 

• Coordination of DSLT (District/School Liaison Team), SAIT 
provider, and school staff to align school plan to address all 
state and federal accountability requirements. 

• Creating and monitoring a plan with SAIT provider to 
implement EPCs 

SAIT providers were perceived as having a positive impact on EPC 
implementation and other aspects of school improvement, especially 
when support was targeted to improving classroom instruction. 

Program: District 
Assistance and 
Invention Team 
(DAIT) 

Study: Westover, 
Strunk, McEachin, 
Smith, Keller, & 
Stump, 2012. Center 
for 

Education & 
Evaluation Services 
at the University of 
California, Davis. 

Mixed methods: 
analysis of California 
Standards Tests (CST) 
student outcome data 
scores in both math and 
ELA in grades 2 to 11 
from 2005-6 through 
2010-11; qualitative 
analysis of surveys were 
administered to DAIT 
providers (for those PI3 
districts with DAITs) 
and district leaders (for 
all PI3 districts in 
Cohorts 1 and 2) and 
interviews were 
conducted with district 
leadership and DAIT 
providers for each 

Cohort 1 and 2 district 
with a DAIT 

Overall, the results were negligible with a few instances 
of slightly positive effects. 

DAIT had a small positive effect on math scores for 
Cohort 1. Students in Cohort 1 districts with DAITs 
(categorized as severe or moderate) outperformed students 
in non-DAIT Cohort 1 districts (categorized as light and 
not required to contract with a DAIT) in math, particularly 
in years two and three. 

DAIT had a slight overall positive effect on ELA scores. 
Students in Cohort 1 districts with DAITs did not have 
significantly higher test results in ELA; however when 
combining results from both cohorts 1 and 2, there is 
evidence of a positive impact on ELA scores. 

There were small positive outcomes for some student 
subgroups. Groups of minority students (African 

American and Elispanic), students who qualify for the free 
and reduced lunch program, and English learners (ELs). 

DAIT providers engaged with their districts as anticipated, including 
conducting initial needs assessment and remaining engaged in assisting 
their districts for an average of two years. 

Areas identified as experiencing the most growth during 
implementation were: data systems and monitoring; curriculum, 
instruction and assessment; governance, and English language 
development (ELD). 

Student achievement growth was associated with district focus on 
data use, high expectations, accountability, recruitment and 
retention policies, and improving instruction. 

Implementation barriers included superintendent turnover, 
contentious relationships at the district-level, lack of financial 
resources, and normalized underachievement. 
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